{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- By: Proskurin Oleksandr\n",
    "- Email: proskurinolexandr@gmail.com\n",
    "- Reference: Advances in Financial Machine Learning, Marcos Lopez De Prado, pg 59-73"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In Chapter 3 notebooks, we have understood how Triple-Barrier and Meta-Labelling concepts work. The next problem in financial machine learning is non-independent samples as a result of that standard machine learning models like Random Forest and Bagging Classifier need to be modified. In this notebook we will tackle the proble of concurrency and the solution to that - __Sequential Bootstrapping__. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "import numpy as np\n",
    "import mlfinlab as ml\n",
    "from numba import jit, prange\n",
    "from mlfinlab.sampling.bootstrapping import get_ind_mat_average_uniqueness, get_ind_matrix, seq_bootstrap\n",
    "from mlfinlab.sampling.concurrent import get_av_uniqueness_from_tripple_barrier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We take dollar bars and barrier events generated from https://github.com/hudson-and-thames/research/blob/master/Chapter3/2019-03-06_JJ_Trend-Follow-Question.ipynb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "barrier_events = pd.read_csv('barrier_events.csv', index_col=0, parse_dates=[0,2]) \n",
    "close_prices = pd.read_csv('official_data/dollar_bars.csv', index_col=0, parse_dates=[0,2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In financial machine learning, samples are not independent. The most part of traditional machine learning algorithms assume that samples are i.i.d, in case of financial machine learning samples are neither identically distributed not indepedent.\n",
    "In this section we will tackle the problem of samples dependency. As you remember, we mostly label our datasets using triple-barrier method. Each label in triple-barrier event has label index and label endtime (t1) which corresponds to time when one of barriers was touched. Let's look at example of 3 samples: A, B, C. \n",
    "\n",
    "Imagine that \n",
    "\n",
    "A was generated at $t_1$ and triggered on $t_8$\n",
    "\n",
    "B was generated at $t_3$ and triggered on $t_6$\n",
    "\n",
    "C was generated on $t_7$ and triggered on $t_9$\n",
    "\n",
    "In this case we see that A used information about returns on $[t_1,t_8]$ to generate label-endtime which overlaps with $[t_3, t_6]$ which was used by B, however C didn't use any returns information which was used by to label other samples. Here we would like to introduce the concept of concurrency. \n",
    "\n",
    "We say that labels $y_i$ and $y_j$ are concurrent at $t$ if they are a function of at least one common return at $r_{t-1,t}$\n",
    "\n",
    "In terms of concurrency label C is the most 'pure' as it doesn't use any piece of information from other labels, while A is the 'dirtiest' as it uses information from both B and C. We can measure average label uniqueness using get_av_uniqueness_from_tripple_barrier function from mlfinlab package"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2019-07-11 14:05:35.694291 100.0% num_concurrent_events done after 0.03 minutes. Remaining 0.0 minutes..\n",
      "2019-07-11 14:05:37.332421 100.0% _get_average_uniqueness done after 0.02 minutes. Remaining 0.0 minutes..\n"
     ]
    }
   ],
   "source": [
    "av_unique = get_av_uniqueness_from_tripple_barrier(barrier_events, close_prices.close, num_threads=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By understanding average label uniqueness you can measure how 'pure' your dataset is based on concurrency of labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tW    0.163135\n",
       "dtype: float64"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "av_unique.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at one of the labels which have average uniqueness equal to 1 which means that it doesn't use any information from other labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "unique_label_index = av_unique[av_unique.tW == 1].index[0] # take the first sample "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Timestamp('2012-02-27 15:08:12.160000')"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "unique_label_index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>side</th>\n",
       "      <th>t1</th>\n",
       "      <th>trgt</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2012-02-27 15:08:12.160</th>\n",
       "      <td>-1.0</td>\n",
       "      <td>2012-02-28 06:13:08.151</td>\n",
       "      <td>0.005140</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2012-03-07 14:52:28.079</th>\n",
       "      <td>-1.0</td>\n",
       "      <td>2012-03-08 09:14:10.272</td>\n",
       "      <td>0.005617</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2012-03-07 16:10:42.766</th>\n",
       "      <td>1.0</td>\n",
       "      <td>2012-03-08 09:14:10.272</td>\n",
       "      <td>0.006731</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2012-03-07 20:48:01.998</th>\n",
       "      <td>1.0</td>\n",
       "      <td>2012-03-08 19:53:12.251</td>\n",
       "      <td>0.008542</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2012-03-08 08:00:15.623</th>\n",
       "      <td>1.0</td>\n",
       "      <td>2012-03-08 20:15:41.320</td>\n",
       "      <td>0.008443</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                         side                      t1      trgt\n",
       "2012-02-27 15:08:12.160  -1.0 2012-02-28 06:13:08.151  0.005140\n",
       "2012-03-07 14:52:28.079  -1.0 2012-03-08 09:14:10.272  0.005617\n",
       "2012-03-07 16:10:42.766   1.0 2012-03-08 09:14:10.272  0.006731\n",
       "2012-03-07 20:48:01.998   1.0 2012-03-08 19:53:12.251  0.008542\n",
       "2012-03-08 08:00:15.623   1.0 2012-03-08 20:15:41.320  0.008443"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "barrier_events[barrier_events.index >= unique_label_index].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see sample 2012-02-27 15:08:12.160 doesn't use any information from 2012-03-07 14:52:28.079 which is the next label.\n",
    "\n",
    "We would like to build our model in such a way that it takes into account labels concurrency. In order to do that we need to look at bootstrapping algorithm of Random Forest.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Bagging, Bootstrapping and Random Forest"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The key power of ensemble learning techniques is bagging (which is bootstrapping with replacement). The key idea behind bagging is to randomly choose samples for each decision tree. In this case trees become diverse and by averaging predictions of diverse tress built on randomly selected samples and random subset of features data scientists make the algorithm much less prone to overfit. \n",
    "\n",
    "However, in our case we would not only like to randomly choose samples but also choose samples which are unique and non-concurrent. But how can we solve this problem? Here comes Sequential Bootstrapping algorithm.\n",
    "\n",
    "The key idea behind Sequential Bootstrapping is to select samples in such a way that on each iteration we maximize average uniqueness of selected subsamples. Let's try to understand how this technique works on toy example from the book."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Consider a set of labels $\\left\\{y_i\\right\\}_{i=0,1,2}$ where: \n",
    "\n",
    "label $y_0$ is a function of return $r_{0,2}$\n",
    "\n",
    "label $y_1$ is a function of return $r_{2,3}$\n",
    "\n",
    "label $y_2$ is a function of return $r_{4,5}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first thing we need to do is to build and indicator matrix. Columns of this matrix correspond to samples and rows correspond to price returns timestamps which were used during samples labelling. In our case indicator matrix is"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "ind_mat = pd.DataFrame(index = range(0,6), columns=range(0,3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "ind_mat.loc[:, 0] = [1, 1, 1, 0, 0, 0]\n",
    "ind_mat.loc[:, 1] = [0, 0, 1, 1, 0, 0]\n",
    "ind_mat.loc[:, 2] = [0, 0, 0, 0, 1, 1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   0  1  2\n",
       "0  1  0  0\n",
       "1  1  0  0\n",
       "2  1  1  0\n",
       "3  0  1  0\n",
       "4  0  0  1\n",
       "5  0  0  1"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ind_mat"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One can use get_ind_matrix method from mlfinlab to build indicator matrix from triple-barrier events."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "triple_barrier_ind_mat = get_ind_matrix(barrier_events)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can get average label uniqueness on indicator matrix using get_ind_mat_average_uniqueness function from mlfinlab."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "ind_mat_uniqueness = get_ind_mat_average_uniqueness(triple_barrier_ind_mat)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's get the first sample average uniqueness (we need to filter out zeros to get unbiased result)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.26886446886446885"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "first_sample = ind_mat_uniqueness[0]\n",
    "first_sample[first_sample > 0].mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tW    0.238776\n",
       "Name: 2011-09-05 14:12:23.480000, dtype: float64"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "av_unique.iloc[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see it is quite close to values generated by __get_av_uniqueness_from_tripple_barrier__ function call. \n",
    "\n",
    "Let's move back to our example. In Sequential Bootstrapping algorithm we start with an empty array of samples ($phi$) and loop through all samples to get the probability of chosing the sample based on average uniqueness of reduced indicator matrix constructed from [previously chosen columns] + sample\n",
    "\n",
    "`\n",
    "phi = []\n",
    "while length(phi) < number of samples to bootstrap:\n",
    "    average_uniqueness_array = []\n",
    "    for sample in samples:\n",
    "        previous_columns  = phi\n",
    "        ind_mat_reduced = ind_mat[previous_columns + i]\n",
    "        average_uniqueness_array[sample] = get_ind_mat_average_uniqueness(ind_mat_reduced)   \n",
    "    // normalise so that probabilities sum up to 1\n",
    "    probability_array = average_uniqueness_array / sum(average_uniqueness_array) \n",
    "    chosen_sample = random_choice(samples, probability = probability_array)\n",
    "    phi.append(chosen_sample)`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For peformance increase we optimized and parallesied for-loop using numba, which corresponds to bootstrap_loop_run function.\n",
    "\n",
    "Not let's finish the example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   0  1  2\n",
       "0  1  0  0\n",
       "1  1  0  0\n",
       "2  1  1  0\n",
       "3  0  1  0\n",
       "4  0  0  1\n",
       "5  0  0  1"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ind_mat"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "to be as close to mlfinlab implementation let's convert ind_mat to numpy matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "ind_mat = ind_mat.values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1st iteration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On the first step all labels will have equal probalities as average uniqueness of matrix with 1 column is 1. Say we have chosen 1 on the first step"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2nd iteration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/alex/anaconda3/lib/python3.7/site-packages/mlfinlab/sampling/bootstrapping.py:50: RuntimeWarning: invalid value encountered in true_divide\n",
      "  average = ind_mat.T / conc\n",
      "/home/alex/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:6: RuntimeWarning: invalid value encountered in greater\n",
      "  \n"
     ]
    }
   ],
   "source": [
    "phi = [1] # Sample chosen from the 2st step\n",
    "uniqueness_array = np.array([None, None, None])\n",
    "for i in range(0, 3):\n",
    "    ind_mat_reduced = ind_mat[:, phi + [i]]\n",
    "    label_uniqueness = get_ind_mat_average_uniqueness(ind_mat_reduced)[-1] # The last value corresponds to appended i\n",
    "    uniqueness_array[i] = (label_uniqueness[label_uniqueness > 0].mean())\n",
    "prob_array = uniqueness_array / sum(uniqueness_array)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.35714285714285715, 0.21428571428571427, 0.42857142857142855],\n",
       "      dtype=object)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prob_array"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Probably the second chosen feature will be 2 (prob_array[2] = 0.42857 which is the largest probability). As you can see up till now the algorithm has chosen two the least concurrent labels (1 and 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3rd iteration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/alex/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:6: RuntimeWarning: invalid value encountered in greater\n",
      "  \n"
     ]
    }
   ],
   "source": [
    "phi = [1,2]\n",
    "uniqueness_array = np.array([None, None, None])\n",
    "for i in range(0, 3):\n",
    "    ind_mat_reduced = ind_mat[:, phi + [i]]\n",
    "    label_uniqueness = get_ind_mat_average_uniqueness(ind_mat_reduced)[-1]\n",
    "    uniqueness_array[i] = (label_uniqueness[label_uniqueness > 0].mean())\n",
    "prob_array = uniqueness_array / sum(uniqueness_array)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.45454545454545453, 0.2727272727272727, 0.2727272727272727],\n",
       "      dtype=object)"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prob_array"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sequential Bootstrapping tries to minimise the probability of repeated samples so as you can see the most probable sample would be 0 with 1 and 2 already selected."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "4th iteration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "phi = [1, 2, 0]\n",
    "uniqueness_array = np.array([None, None, None])\n",
    "for i in range(0, 3):\n",
    "    ind_mat_reduced = ind_mat[:, phi + [i]]\n",
    "    label_uniqueness = get_ind_mat_average_uniqueness(ind_mat_reduced)[-1]\n",
    "    uniqueness_array[i] = (label_uniqueness[label_uniqueness > 0].mean())\n",
    "prob_array = uniqueness_array / sum(uniqueness_array)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.32653061224489793, 0.3061224489795918, 0.36734693877551017],\n",
       "      dtype=object)"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prob_array"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The most probable sample would be 2 in this case"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After 4 steps of sequential bootstrapping our drawn samples are [1,2,0,2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see how this example is solved by mlfinlab implementation. To reproduce that:\n",
    "\n",
    "1) we need to set warmup to [1], which corresponds to phi = [1] on the first step\n",
    "\n",
    "2) verbose = True to print updated probabilities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.33333333 0.33333333 0.33333333]\n",
      "[0.35714286 0.21428571 0.42857143]\n",
      "[0.45454545 0.27272727 0.27272727]\n",
      "[0.32653061 0.30612245 0.36734694]\n"
     ]
    }
   ],
   "source": [
    "samples = seq_bootstrap(ind_mat, sample_length=4, warmup_samples=[1], verbose=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1, 2, 0, 2]"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "samples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see the first 2 iterations of algorithm yield the same probabilities, however sometimes the algorithm randomly chooses not the 2 sample on 2nd iteration that is why further probabilities are different from the example above. However, if you repeat the process several times you'll see that on average drawn sample equal to the one from the example "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Monte-Carlo experiment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see how sequential bootstrapping increases average label uniqueness on this example by generating 3 samples using sequential bootstrapping and 3 samples using standard random choise, repeat the experiment 10000 times and record corresponding label uniqueness in each experiment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/alex/anaconda3/lib/python3.7/site-packages/mlfinlab/sampling/bootstrapping.py:50: RuntimeWarning: invalid value encountered in true_divide\n",
      "  average = ind_mat.T / conc\n",
      "/home/alex/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:11: RuntimeWarning: invalid value encountered in greater\n",
      "  # This is added back by InteractiveShellApp.init_path()\n",
      "/home/alex/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:8: RuntimeWarning: invalid value encountered in greater\n",
      "  \n"
     ]
    }
   ],
   "source": [
    "standard_unq_array = np.zeros(10000) * np.nan # Array of random sampling uniqueness\n",
    "seq_unq_array = np.zeros(10000) * np.nan # Array of Sequential Bootstapping uniqueness\n",
    "for i in range(0, 10000):\n",
    "    bootstrapped_samples = seq_bootstrap(ind_mat, sample_length=3)\n",
    "    random_samples = np.random.choice(ind_mat.shape[1], size=3)\n",
    "    \n",
    "    random_unq = get_ind_mat_average_uniqueness(ind_mat[:, random_samples])\n",
    "    random_unq_mean = random_unq[random_unq > 0].mean()\n",
    "\n",
    "    sequential_unq = get_ind_mat_average_uniqueness(ind_mat[:, bootstrapped_samples])\n",
    "    sequential_unq_mean = sequential_unq[sequential_unq > 0].mean()\n",
    "    \n",
    "    standard_unq_array[i] = random_unq_mean\n",
    "    seq_unq_array[i] = sequential_unq_mean"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.6666666666666666, 0.6666666666666666)"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.median(standard_unq_array), np.median(seq_unq_array)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.6430261904761904, 0.7036232142857142)"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.mean(standard_unq_array), np.mean(seq_unq_array)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "KDE plots of label uniqueness support the fact taht sequential bootstrapping gives higher average label uniqueness"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x7f5579410b38>"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAW4AAAD8CAYAAABXe05zAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzsvXl4m2eZ7/95tdhavMjy7tiOnbVZHGdrmqQtaaG0Q0kLQ1sKB5ihBTocoB04ZzjDGWDgMMz8YGCYYRg4ULYWTgcY2kJpoW2aNmmb7lnbxEm8xYnteLcla7ek9/n98VryJtuyLcmW/HyuK5di6dX73ralr299n/u+H0UIgUQikUjSB91iByCRSCSSuSGFWyKRSNIMKdwSiUSSZkjhlkgkkjRDCrdEIpGkGVK4JRKJJM2Qwi2RSCRphhRuiUQiSTOkcEskEkmaYUjGSYuKikRNTU0yTi2RSCQZybFjx/qFEMXxHJsU4a6pqeHo0aPJOLVEIpFkJIqiXIz3WGmVSCQSSZohhVsikUjSDCncEolEkmYkxeOORTAYpKOjA7/fn6pLStIQk8lEZWUlRqNxsUORSJYsKRPujo4OcnNzqampQVGUVF1WkkYIIRgYGKCjo4Pa2trFDkciWbKkzCrx+/0UFhZK0ZZMi6IoFBYWyk9lEskspNTjlqItmQ35GpFIZkcuTkokksxiGWzHuKyEW6/Xs3XrVjZv3swtt9yCw+FIyHnb2trYvHlzQs41nvPnz3PdddexdetWNmzYwD333JPwa4zn8OHD7N+/H4A//OEPfOMb30jq9SSShDJ8GX5yA/z+k4sdSdJZVsJtNps5efIkp0+fxm638/3vf3+xQ5qR++67j8997nOcPHmSs2fPcu+996bs2rfeeitf+MIXUnY9iWRBXD4JP74eOt6A049CwL3YESWVZSXc49mzZw+dnZ0AuN1u3vGOd7B9+3bq6up47LHHAC2T3rBhA5/4xCfYtGkTN954Iz6fD4Bjx45RX1/Pnj17JvwB8Pv93HXXXdTV1bFt2zYOHToEwAMPPMB73/tebrnlFmpra/mP//gPvvOd77Bt2zZ2797N4ODglBi7urqorKyMfl1XVxeN69prr2X79u1s376dl19+GdAy5n379vH+97+fdevW8YUvfIGHHnqIXbt2UVdXR0tLCwAf/ehH+eQnP8m1117LunXreOKJJ6Zc+4EHHuAzn/lM9Pj77ruPvXv3smrVKh5++GEAVFXlU5/6FJs2bWL//v3cfPPN0cckkpTh7oMHbgZVhV33QHgELjy/2FEllZSVA47n/zx+hobLwwk958aKPL5yy6a4jg2Hwzz77LN87GMfA7Ta4d/97nfk5eXR39/P7t27ufXWWwFoamriV7/6FT/+8Y95//vfzyOPPMKHP/xh7rrrLr73ve+xb98+Pv/5z0fPHRHxt956i3PnznHjjTfS2NgIwOnTpzlx4gR+v581a9bwzW9+kxMnTvC5z32OX/ziF3z2s5+dEOfnPvc53v72t7N3715uvPFG7rrrLmw2GyUlJTzzzDOYTCaampr44Ac/GJ0Nc+rUKc6ePYvdbmfVqlV8/OMf5/XXX+e73/0u3/ve9/i3f/s3QBP/559/npaWFq6//nqam5tn/Jl1dXVx5MgRzp07x6233srtt9/Oo48+SltbG2+99Ra9vb1s2LCBu+++O67fgUSSMNpfgxEPvOMrULQOTjwEjU/BFe9e7MiSRlwZt6IoNkVRHlYU5ZyiKGcVRdmT7MCSgc/nY+vWrRQWFjI4OMg73/lOQKsf/ru/+zu2bNnCDTfcQGdnJz09PQDU1taydetWAHbs2EFbWxtOpxOHw8G+ffsA+MhHPhK9xpEjR6JfX3HFFaxcuTIq3Ndffz25ubkUFxeTn5/PLbfcAmiZdFtb25R477rrLs6ePcsdd9zB4cOH2b17N4FAgGAwyCc+8Qnq6uq44447aGhoiD7nyiuvpLy8nOzsbFavXs2NN94Y8xrvf//70el0rF27llWrVnHu3LkZf3bvfe970el0bNy4MfqzOXLkCHfccQc6nY6ysjKuv/76+H4REkkiuXwCFD3YV4POABVboelARi9Sxptxfxd4Sghxu6IoWYBlIReNNzNONBGP2+l0sn//fr7//e9z33338dBDD9HX18exY8cwGo3U1NREa4mzs7Ojz9fr9fh8PoQQ05atiRleLOPPpdPpol/rdDpCoVDM51RUVHD33Xdz9913s3nzZk6fPs3jjz9OaWkpp06dQlVVTCbTnK8xOf7ZyvDGnzfyPc70vUokKaPrJNiqwTD6Gq3cBRdfgu43obx+cWNLErNm3Iqi5AFvA34KIIQYEUIkphxjkcjPz+ff//3f+fa3v00wGMTpdFJSUoLRaOTQoUNcvDjzdEWbzUZ+fj5HjhwB4KGHHoo+9ra3vS36dWNjI5cuXWL9+vXzivOpp54iGAwC0N3dzcDAACtWrMDpdFJeXo5Op+OXv/wl4XB4zuf+7W9/i6qqtLS00NraOq8Yr7nmGh555BFUVaWnp4fDhw/P+RwSyYIQAjqPQ+GasftW7AAUaHx60cJKNvFYJauAPuDniqKcUBTlJ4qiWJMcV9LZtm0b9fX1/PrXv+ZDH/oQR48eZefOnTz00ENcccUVsz7/5z//OZ/+9KfZs2cPZrM5ev+nPvUpwuEwdXV13HnnnTzwwAMTstW5cODAATZv3kx9fT033XQT3/rWtygrK+NTn/oUDz74ILt376axsRGrde6/jvXr17Nv3z7e9a538cMf/nBC1h4vt912G5WVlWzevJm/+qu/4qqrriI/P3/O55FI5o2zA3yDULh27D6zTfO6M1i4ldk+7iqKshN4FbhaCPGaoijfBYaFEF+edNw9wD0A1dXVOyZnrWfPnmXDhg2JjF0yTz760Y+yf/9+br/99gWfy+12k5OTw8DAALt27eKll16irKxsQeeUrxVJ3DT8Af7rI3Dzv0DxuE+Np34FJ/8TPt8C1sLFi28OKIpyTAixM55j4/G4O4AOIcRro18/DEwp8BVC3A/cD7Bz505pfi4T9u/fj8PhYGRkhC9/+csLFm2JZE50nQSdHuyThpIVXwEI6DsL1msWJbRkMqtwCyG6FUVpVxRlvRDiPPAOoGG250mWLg888EDCziV9bcmicvkE2FaCPmvi/bnl2u3gBahZhsI9yr3AQ6MVJa3AXckLSSKRSOJACE24K6+c+pi1WCsNHGxNfVwpIC7hFkKcBOLyXiQSiSQlOC6Bb2hiRUkEnR5ySjNWuJdty7tEIklzLp/QbsdXlIwnt0wKt0QikSwput/UMuuCmtiP51Zowp2BjWLLSrj/8R//kU2bNrFlyxa2bt3Ka6+9NvuTkszhw4ejQ6IAfvjDH/KLX/xixud89atf5dvf/vaU++UYWMmyYugiWEtBP83+pHnlMOIG70Bq40oBizJkajF45ZVXeOKJJzh+/DjZ2dn09/czMjKy2GFx+PBhcnJy2Lt3LwCf/OT8ZwlHxsC+5z3vAbRBV6ni1ltvjQ7mkkhSguMSWIumfzxaWdI683FpyLLJuLu6uigqKop2MRYVFVFRUQFoI1r37dvHjh07uOmmm+jq6oreHxnd+vnPfz66WcL4kaeg1TJHyuIOHDjAnj172L59O3fccQdutzYXuKamhq985SvR0bHnzp2jra2NH/7wh/zrv/4rW7du5cUXX5yQTf/4xz/myiuvpL6+nttuuw2v1zvr9yjHwEqWDc52sJZM/3iu9v7ORJ97cTLuJ78A3QnOBsvq4F3Tf1S/8cYb+drXvsa6deu44YYbuPPOO9m3bx/BYJB7772Xxx57jOLiYn7zm9/wxS9+kZ/97GfTjm6djv7+fr7+9a9z8OBBrFYr3/zmN/nOd77D3//93wPaH4vjx4/zgx/8gG9/+9v85Cc/4ZOf/CQ5OTn8zd/8DQDPPvts9Hzve9/7+MQnPgHAl770JX7605/OuJmCHAMrWTaEg+DqhlXXTX9MTgkoOinc6UxOTg7Hjh3jxRdf5NChQ9x555184xvfYOfOnZw+fTo64jUcDlNeXh5zdOuTTz454zVeffVVGhoauPrqqwEYGRlhz56xCbjve9/7AG087KOPPjprzKdPn+ZLX/oSDocDt9vNTTfdNOPxd911FzfddBNPPfUUjz32GD/60Y84deoUwWCQz3zmM5w8eRK9Xh8dMwtjY2CBKWNgI5tAgBwDK1liDHcCQqvXng69UcvIpXAniBky42Si1+u57rrruO6666irq+PBBx9kx44dbNq0iVdeeWXCsQ6HY9pRpwaDAVVVo19HRsAKIXjnO9/Jr371q5jPi9g0er1+2jGu4/noRz/K73//e+rr63nggQfi6lKUY2AlywJnh3Y7k1UCWkngQOYJ97LxuM+fP09TU1P065MnT7Jy5UrWr19PX19fVLiDwSBnzpyZcXRrTU0NJ0+eRFVV2tvbef311wHYvXs3L730UtRG8Hq9E7LbWOTm5uJyuWI+5nK5KC8vJxgMTrj+dMgxsJJlg6Ndu82ZIeMGbYFyKPOEe9lYJW63m3vvvReHw4HBYGDNmjXcf//9ZGVl8fDDD3PffffhdDoJhUJ89rOfZdOmTfz85z/n7rvvxmKxTLAprr76ampra6mrq2Pz5s1s374dgOLiYh544AE++MEPEggEAPj617/OunXrpo3rlltu4fbbb+exxx7je9/73oTH/uEf/oGrrrqKlStXUldXN63ARzhw4AB//dd/Hc2ox4+Bve222/jtb3/L9ddfv6AxsD09PQsaA/vss8+yefNm1q1bJ8fASuZPNOOeRbjzyrXuSt8QmAuSH1eKmHWs63zYuXOniCx+RUj3UZ1tbW3s37+f06dPL3YoKSfVY2DT/bUiSQF/uBfOPg7v/+XMx116BQ79I3ziEKzYnprY5kmix7pKJAlDjoGVJARH++zZNkys5V7iwj0XpHDHSU1NzbLMtkGOgZUsQZzt2hCp2cgdTQwGLyQ3nhST0sVJWVUgmQ35GpHMihCaxz1bRQmAwaR528MdyY8rhaRMuE0mEwMDA/KNKZkWIQQDAwPzWviULCO8AxDyz15REsFcoDXrZBAps0oqKyvp6Oigr68vVZeUpCEmk2lC275EMgXHJe02nowbwGwHV1fy4lkEUibcRqOR2tra2Q+USCSSmYi3FDCCxQ6XTyYvnkVg2TTgSCSSDMEZab6JM+O2FIKnD8KzdyunC1K4JRJJeuFo1xYds3LiO95sB4Qm3hmCFG6JRJJeONtHJ//NPC8nitmu3WaQzy2FWyKRpBfOOJtvIlgiwp05lSVSuCWSFKGqgm89fY7vH5p5lrlkFuLtmoxgybyMW3ZOSiQpIKwK/vaRN3n4WAcmo467r67FnKVf7LDSj3AQfINj9kc8mGzahgoy45ZIJHMhIto7VhbgD6q82JQ5C2UpxdOv3Zpt8T9Hpx9twsmcjFsKt0SSZPpcAR4+1sGfbS7jszesxZql55mGnsUOKz3x9Gq3pjkIN2gZujtzfuZxWSWKorQBLiAMhOIdPSiRSOBkuwOAq2rtGHQ66qtsPHuul7Aq0OvirIyQaLhHP6nMdba22QbDyzPjvl4IsVWKtkQyN061O9ApUFukbWCxc2UBg54Rjl8aWuTI0pBoxj3HDTgshdIqkUgk8XOifYiVhVayDdpiZH2VDYNOkXbJfIg00czF4wbNKvH2a4ubGUC8wi2AA4qiHFMU5Z5YByiKco+iKEcVRTkqB0lJJBqqKjjV7mR18dh2cZYsAxsr8jhwpltOy5wr7l7QZ4PBPLfnRUoCM8Tnjle4rxZCbAfeBXxaUZS3TT5ACHG/EGKnEGJncfEcaiwlkgymtd+NOxBidfHE9uz6ShttA156XYFFiixN8fRp2fakrklVCP6/V/2cHZhmI2xLoXabISWBcQm3EOLy6G0v8DtgVzKDkkgyhROXtIXJNSUThbuyQMsYW/s8KY8prXH3xvS3z/Sr/OjUCJ942oszEONTjDmzuidnFW5FUayKouRG/g/cCCzPPbwkkjlyqsOBJUtPhW3iR/vI1xf6pXDPCXdPzFLA17q0yX+X3YK/fd431YLKsO7JeDLuUuCIoiingNeBPwohnkpuWBJJZnDikoNVxVZ0kz7a261ZZOl1tPa5FymyNMXTH3Nh8tXLYcqt8NEN8NSFEP95dtIiZHYeKPrlk3ELIVqFEPWj/zYJIf4xFYFJJOmOPxjmfLeLNcVTx4/qFIWyfJPMuOeCqmrblpkm1nCrQvBGV4jNdvjzVbAmHx4+PzLxudHuyWUi3BKJZH6cuewkpIopC5MRyvJNtErhjh/fIIjwlIz73KCKcwTqCkGnwDobtDrU2HbJMrJKJBLJPGjp1US5ym6J+XhFvon2QS/BsJrKsNIXd+zmm9cua/523WjhSGUOOEdgwD9JuDNo70kp3BJJkrgw4EGvUyjKyY75eFm+mZAq6BjypTiyNGWa5pvXusKUmqFk9O/jitEPOK2OSX8QM2jQlBRuiSRJXBzwUJqbPe08kvJ8E4BcoIyXiHCP87iFELzWFWJz4dhhVaPC3RJLuH1DGdE9KYVbIkkSF/o8lOSZpn08ItxygTJOIlbJuIy7aUhlyM8E4S42Q5YulnCPPi8yGjaNkcItkSQBIQRtA17K8qcX7lyTkVyTQS5QxounVyvpG7dJ8Jt9WqfkhnGFJjpFs0umWCWR+u8M2DRYCrdEkgT6XAF8wTBlM2TcoGXdF2T3ZHy4p7a7d7q1BciySeu/lTnQ7JjU/h7NuHuTGWVKkMItkSSBtgEvwKzCXZZnorVfetxx4emd0jXZ5VaxZ4Nx0i5wlTnQ4RIEwuMqSyLPdcuMWyKRxKBt1P6YySoBKLeZ6RkO4AmEUhFWeuPumVJR0ulWKYoxKLDSCqqAi85xdom0SiQSyUy0zVIKGEEuUM4BT9+UjPuyW1AcS7hjVZYYzaDPklaJRCKJTduAh5IZSgEjlOfLYVNxIcSUOSVCCC671ZjCHbOWW1G0kkBplUgkklhc6PdQOou/DVA8mpF3OmQTzoz4nRAemdA16QgIfCFiCrfZAEXmGCWBpnxplUgkkqkIIWjrn7kUMII5S09OtoFO2T05M5Ha63HNN5dHK0piCTdoPveUyhJTvrRKJBLJVPrc8ZUCRijKyZIZ92x4pjbfXHZr2XTJdMKdE2PYlLlgrJEnjZHCLZEkmLb++EoBIxTlZNM+6E1mSOlPdMDUeOHWBDlWVQlos0vcQXCNn/BqytdGw6rpPdjLsNgBSCSZRtvAxFJAXdBNXu9RLIMN6ENeOrbch9BnRY8vysmmoWsYIQSKMvNi5rIlxoCpy24Vow7ys2I/xT5a0NPrVcnLHi30NtlADYHfMbYrThoihVsiSTBt/WOlgLqgly1/vBWzqy36uD7opm3XV6NfF+dm4x0J4/QFsVmmUaHljrsXULSdbEbpHK0oma5wxz76gafXK1gTscYjwu/uTWvhllaJRJJgLg54o6WAVSf/BbOrjfbNn+HsdT+mv/pmys//gqKWR6PHR2q95XjXGfD0gSlP28lmlOlKASMUjGbcfd4Y3ZNpvkAphVsiSTCt/W5K80zk9B2n/NwDDFbewHD5XlSjlZ61H8RdsJHVr30Ry+BZQFucBFkSOCPuqe3u0zXfRBjLuDOve1IKt0SSQIQQXBzwUpmrY/XLf0vQZKdn7QfGDtDp6dhyL6rOyIrTPwCgKHe0lltm3NMzqd09GBb0egVFM6z/Wg2QrdeskijmzJhXIoVbIkkgfe4A3pEwbw+/hGW4ha4rPopqmDi6LpyVj7P8GuztT2PwD5KbbSDboJMZ90xMGjDV4xWoYvpSQNAaJe3Z0DM+487O1UbDSqtEIpFEuDg6FXC3448ELGW4i7bHPG6o4jp0aoiiC79HURSKc7Nlxj0Tnv5JpYCaGM9klQAUmCZl3IouI7onpXBLJAnkQr+HWqWLCudxHBX7JsyOHk8gtxpv3mpKm/4LhKAwJ4tOh6zljsmIB4LeSaWAM3dNRijIhl7P5E2DbdIqkUgkY1wc8HCn4XmEosNR/rYZj3WsuA6Ls5GcgTcpsmbLqpLpiO41OfeM226atDgJGdH2HrdwK4qiVxTlhKIoTyQzIIkknbnUN8zt+hdwFW0jNG6uRiycZXtQ9dmUNP8XRbnZDHmDeEfkXO4puGM33+RmgWmWThR7ttY96QtOKglcRhn3XwNnkxWIRJIJFHc9TxEOHBXXzXqsarAwXHIlhW1/pMSq1SdLnzsG0TklY38Ie72CwplHnQOaxx05PorJpp1TiNhPSgPiEm5FUSqBdwM/SW44Ekn6IoTgKvczDOvycRVtjes5ruLtGILDbAg3AtAhK0umErVKxka69vtU8uMQ7oi4T6gsMdsg5IeR9N0yLt6M+9+A/wVMO5lFUZR7FEU5qijK0b6+9P4YIpHMh37HMNdwkuacnRM6/GbCba9DKDrWDL8KyIw7Ju6pHnefV0Q7I2fCPl3GDWk9JXBW4VYUZT/QK4Q4NtNxQoj7hRA7hRA7i4uLExagRJIuDJ1+BqsSYKBwZ9zPUY1WvPlrqeg7gl6nyFruWHh6ISsH9MboXQM+Me1wqfEUjBs0FSW623t/AoNMLfFk3FcDtyqK0gb8Gni7oij/L6lRSSRpiKHxT7iEGaV045ye5y6sJ2fwNGutPplxx2JSu7svKPCGwBZHxp2XBQZlmow7jStLZhVuIcT/FkJUCiFqgA8AzwkhPpz0yCSSdEINU9r1HM+r9ZRYjbMfPw530RYAbsg6LTPuWLh7J1SU9Ps1EY5HuBUlRhNOxCvPZKtEIpHEQcdRrKEh3jDsRD/Hd5U/t4ZgVj57xUk6hmQTzhQ8vRMXJkdtj3iEGzS7pG8ZWiVRhBCHhRD7kxWMRJK2nP8jQfS0WuOrJpmAosNTWEd94Cj9wz5GQum9O0vC8fRNsEoGfPFn3KDVcneP757UGbS53plslUgkktkRZ5/gdXUjhXmW2Q+OgatwK9bwMJuVVrqd/gRHl8aERrQd3sdbJRHhjnPPidjdkzZplUgky5qBFpTBFp4Ob6c6Z36n8BRuAmCProEOObNkjOiWZWPNN3POuE3gDEAgPMnnXi5WiUQiiUHLcwAcVrdSnTu/U4Sz8nFbKtmja5CVJePxTN0kuM8nsBogK75S+ejek32T53JLq0QiWcY0P4Mjq5xLopSqeQo3gM++gZ2683QNuhIXW7oTyYrn0TUZoSDmTjj50iqRSJYtoQBceIG3suopMkHO3CoBJ+C3b8SqBKDrZOLiS3fcU+eUDPhE3DYJzLD3ZGBY+/2lIVK4JZKFcOkVCPp4LlRH1Tz97QiegisAKO5/IwGBZQixrBJvfF2TESIiP+iPsYVZmm6oIIVbIlkIzQcROiN/cG+Yt78dIZyVT4e+klWeE4mJLRNw94HBBMaxzSUH/Gpcc0oiREQ+sqgJpP28EincEslCaDpIoHAjAyHTgoUboM20kc3hBtTgyMJPlglM2msypAocfubkcWfpwWKYJNwy45ZIlinOTug7S3uu1nSTCOHuz9V87qHm1xZ+skzA0wfmsYXJQb9AEH8pYIT8bC1Tj2KSwi2RLE9angXguCFxwu2zaz63v/n5hZ8sE3D3wLidhCLNNwVz8LhBs0sGpVUikUhoPgiWIl73raBwgRUlEWx5eZxTqzBeennhJ8sE3H0TuiYjdsdcrBLQhLt/vHAbTWAwy4xbIllWhEPQcghWbKfRoS64oiRCiRleUTdSMHBMa/dezqhh8A1O8LgjA6bmsjgJo1aJL8Zu71K4JZJlROcxCAyjVmyneUhlZQJsEtA2v31TtwGj6ofLy7y6xDsAQp3QfDPgn1/GbcuCIb9AHb/PpLlAWiUSybKi+SAoejqsW/CFWFDH5GQumjSfm7YXE3fSdCTGnJI+r8CoA+ssu7tPJj8bQgKGx/fbZOfJjFsiWVY0H4Ti9bwxpNUXbyiY5fg5kG3Jo1WphrYjiTtpOuKe2nzT7xPkZ2sbJMyFSC13v2/SXG4p3BLJMsHTr9kYFds42h3GakxMRUmEMgu8GNqAaH91efvc0Yx74uLkXCtKYJruSZNNs2PU8AKCXBykcEskc6XlECBgxQ6OdoXZUAC6OWaAM1GdCy+HN6IEfXD5eOJOnG5E55SMnww4twFTEWJ3T+ZrHrp3YAFBLg5SuCWSudJ8EEz5OKyraXKoCbVJQBPu11Tpc+PpBZ0RjNboXfHu7j6ZiNgP+CctTkJaLlBK4ZZI5oKqQstBKN/KsV7NL91kT+wlqnPAQS79ppXL2+f29GvZ9qihLYSY82TACHkzzStJQ59bCrdEMhe639QEZcUO3ugOY1BgrW32p80FixGKzXDGsAkuvZq2o0cXjHviJsHuIATVuZcCAhh1kGuEgQmLk6PnlsItkWQ4zQe129GFydU2rfY60VTlwPOhjRDyQ/vrib9AOuCeOGAq0rI+H6sEYjThmKRVIpEsD5oPQuEaAlk23uwLszHB/naElbnwmOsKhKKPbo227PD0TtokWMuW5y3cWZM87iyrtuO7zLglkgzG74SON6BiO6f7woyEYWOC/e0IVbkwELYQsF+xPIVbCM2SGp9xz7NrMkJe1qSMW1G0BUop3BJJBnPhBVBDsGI7z7eHUEiecEda6Dtz66HrVFrvSD4v/A5Qg7GFe54Zty3WvBKTLTOtEkVRTIqivK4oyilFUc4oivJ/UhGYRLLkaD4IRiuiaD2PtwSpK5r7XOh4iQytOmncAghoPZycCy1V3LGbb2BhVsmQXxBWJ1WWZGjGHQDeLoSoB7YCf6Yoyu7khiWRLDGEgKaDUF5Pw5COC07B2yqSdzmrEYpM8LK/BrJzR5t+lhGeqc03g36BST//xeD8bBCAIzBeuPPGrpVGzCrcQsM9+qVx9J+Y4SkSSebR3wjDHbBiO4+3BNErsLcsuZeszoXzQ0BZvbZpg1hGb7sYc0oGRueUzBdbrFpus02zodLsZxuXx60oil5RlJNAL/CMEELuqyRZXjQdAEBUbOfx5iDbiue/SBYvVTnQ4lBRK7aCqwv6zif3gkuJiKc/QbjVaCPNfIjZPWmyQXhEW3iKtAIKAAAgAElEQVROI+ISbiFEWAixFagEdimKsnnyMYqi3KMoylFFUY729aWfZySRzEjTAbDVcMJjp9OdXJskwspc8IWgK2+bdkekhnw54OkFRafZRKMM+OfX7h5hxt3e08znnlNViRDCARwG/izGY/cLIXYKIXYWFxcnKDyJZAngH4aLr0DlDv7QHMKogz1JtkkA1o/WiL8wVAC2Gjj/p+RfdKkQ6ZrU6aN3Dc5zTkmEyEJyzN3e06yyJJ6qkmJFUWyj/zcDNwDnkh2YRLJkuPA8qEECpTt4tHGEPWVaW3qyWZkLpRZ49mIIqq6CS6+AdzD5F14KePom2CRCCAb9C/O4c7NAYVLbezTjzjDhBsqBQ4qivAm8geZxP5HcsCSSJUTTATBaeWx4DcMj8O6a1FxWUeDKEjjSESKwYpc2grTx6dRcfLGZ1O7uCUIgPP9SQAC9MtqE44+RcadZnXw8VSVvCiG2CSG2CCE2CyG+lorAJJIlgRDQ9AyiYisPNqjU5CZ+GuBMXFUK/jC85KsBSyGc/2PqLr6YeHrHhkCx8OabCFPmlWTnAUrmWSUSybKm5wy4uriYu50zAyo318x926yFUFcIZgMcvKRC5S5ofhaC/tQFsFhMskoi9sZCK3nys8aGVQGah27Kz0irRCJZvoyWAf58sA6LAa6vTO3ljXrYVqT53KLqKgh6Nc89kwm4Ieib0nwDCci4s7R9KydgLsg8q0QiWdY0HSBcsIZfXczh7ZVgScII19nYVQY9XkGDcTMYLZlfXRLJfk1joxej7e4LzLht2TDoVyfeacqTVolEkjH4hqD9dc6atzIShhuqFieMK0u0hbVvHA0TrtgO555Iyw1u48bVo92axwl3AjNuRwCC4cnzSqRwSySZQcshEGF+7dpCVQ6syZ/9KcnAlg2f3gIvdoT5keNK8PTja3qeAXcAkWat2nHh7tZuxwn3oE+QpQOTfprnxEkkYx+a3D2ZZg04i/DBTyJJE5qeIZyVy3/2reJD61O7KDmZm6q1bbv+/a0t/EW2iT/88t/5u5APuzWL+sp8/uG9m6kssCxegIkkknFbxsp3BvzaXpML/R1Euyf9gpLIHsTmfBjxwIgXstLjZygzbokkFqoKzQdoMdejokv5omQs9tfAX2/P5qx5B+/NPsZfXrWCLSvyebV1gC888lbmZN/ubq3aY1y7+0K7JiPE7J5MwyYcKdySeaGqgmfP9uDyBxc7lOTQdRI8/TzirWeTXetgXAq8bQWUXLEbS3iYDxS28Ff7VvOBK6s50tzPYycvL3Z4icHVA2a7NqtklAH/wgZMRRifcUeJWDLu9LFLpHBL5sWjJzr52INHufafD3H/Cy0EQhm2WNb0DAKF37rqlkS2PR5P4RZCBiuFbVoD8w0bSllTksPXnmjA4R1Z5OgSgLt7gr8NCx/pGiE6IXBC23v67fYuhVsyZ4QQ/OzIBcrzTay0W/inP53jO880LnZYiaXpAD3mNQySl5KBUnNB6Iy4Sq6k8NLTKOEAOp3Cx6+pxeEd4QeHWxY7vIXjmircibJKcoygU6RVIlmGvHZhkIauYd69pZwvvGsD26psPHGqK4M81j7oPMZhdStrbcnbnmwhOMv2oA95KOg8DMDKQiv1lTaePtO9uIElgknC7Q0K/OHEzD/XKdqGCoMxJwTKjDttyRjxSSI/O3KBXJOBa9do43uvrLHT6fDR0DWc8lj8wTDfP9RMS5979oPjpfEpQPBL1zZ2LNEJxZ6CjYSy8ihsG5tdsrXKxsUBLxf6PYsY2QIJjYBvUPO4R0lUDXeE/GzoH+9x67PAaJVWSTrSO+znnl8cZc83nsPpzdAFtwRwacDLMw09vOOKErIM2stn+8oCFODAmZ6UxtI77OfOH73Ct54+z6ceOs5ISJ39SfFw/k/4sos5o65kR0liTplwdHqGS3ZR0HEQXdALQH2VljkePp8+H/mnELErxpcCLnCT4MnkZcGgb9JrxZxeTThSuIHnzvXwzn99gUPne+l2+vnZSxcWO6Qly8PH2lEUeOfGMeM332xkfVkuBxpS9zH9ssPHLf9xhPM9Lm6uK+d8t4v/mwh/d8QLLc9xPGsHVqPCetvsT1ksnGV70If9FHQ8C0BpnokKm4lD59JHgKYQo2uy36uJbKIsK1t2jHklJpu0StIJIQRf/N1pck0GvvG+LVxZU8DPXrqA0yez7lgcvThETaEVu3Vi+rNjZQFnu1y0D3pTEsd/HW2ndzjA3+/fxEd2r2Tv6kK+91wT57tdCztx62EI+fm1extbi0C/hN8hXtt6gtkFFF0cG4+/tdLGq62D+EbStMon2jU5lnFHRLYgQcKdnzVpcRLSbkLgEn5ZpoYzl4fpcvrZv6WcCpuZP99Wicsf4sGX2xY7tCVHWBWc6nCwuiRnymNX1mhvtAMNqbFLnj7TzfqyXGqLtPa3v9xTgyVLzz/+sWFhJz7/J8IGC0/5NrBzqdokERQdw6W7sXUeRj+irS/UV9kYCau80ppe0+6iuKa2u0eEO1EZd34WuIMQCE9aoJQed/pwoKEHnQLbqrUXSm2RlR3VBfz0yIXMbS6ZJ829bjyBMGtjCHdpnolqu4VnUmCXtA96OdvlYsfKsTd3ntnIdetLeLllYP5rFGoYGp+kLWcbQQxsX6ILk+Nxlu1Bpwaxtz8DwIbyPLINOg6dSx8RmoC7B1AmCHefV8VqhKwFzimJEKlOGZxcEugbgnB6vOelcI9mbnmmsU0Eb91agdMX5ODZ1C62LXVOtg8BsKZ4qnADbCzP41S7k7Ca3MqcSMlbJMuPsHNlASFVcGi+i3Odx8DTzzPqDqpyoMi80EiTjy9vNSOm4mgzjlGvY1NFPs83pqlwu7o1ER23SXC/TyTMJgGtHBCm2TQ4TbLuZS3c7YNeznW72FE9UQDWFOdgydJztG1okSJbmpy45CAn20BZvinm4zVFVnzBcNLL0Q6c6aHabqE0b2Icq0tysFmMPDNfu6bhMYTOwINDdWwuTECgqUBRcJbtJr/rCAa/tpHw2tIcLg1607M6yt0DlonNN30Jar6JEO2enND2PqoBrq7EXSiJLGvhjvixO2smvlB0OoW1pTm80bZMdtSOkxOXHKwutqJMM6It4jefuexMWgwD7gBHLw6yc2XBlMd0isL26gIOne+dewu+qsLpR3EVbaMraE0f4QaGy/agE2Hsl7SNhGsLR38PXcn7PSSNGF2T/V6R0Cao6LyS8SWB1iLtdlgK95LnwJluquzmKZkbwPrSPBp73OmZtSQBdyBEY4+LNSW50x5TYTNh1Cuc7kyeYDx7thdVwM6a2Dv27lhZgHckzCstA3M7ccfr4LrMcfMeQNvrMV3w56wkYKmgqO1xYNwf0M7UN0QtmFjC7VMTa5VEPO7xGbdl9Bc+nB6DupatcLv8Qd5oG2RH9dTMDWB9mSZQxy7JrBvgzQ4HAlgTY2EygkGno9pu4XQSBePF5n7s1ixqCmOP69tckY/JqJu7XXL6UdBn8bB3OxVWKIztBi1NRu2SvJ7XMHp7yTMbKcrJ4nQSP/kkBTWseczjSgEDYcHwSOJKAUHbfs6om1TLbcoHnQGGOxN3oSSybIW74fIwqhgT6MmsLrai1ym8IX1uQLNJYPqFyQg1hVbOXHYmbXTAqXYHa4pzprVrsgw6tqyw8UxDT/wxqGFo+D1ixQ5e7MliU+xkfknjLN2DgqDw0pOA9nt4K4mffJKCdxBEeOKWZQkuBQRtM4YptdyKTsu6pce9tIm8qGsKrTEfzzboqS2y8sYFmXEDnGx3UJFvIsc086ZJtUVWhv0hOoZ8CY/B6Q1yadBLbXHs31mEbdU2el0BzvfE2Yxz6RVw99BZdDXOkfSySSKM5KzAl1NN4ahdUlNk5UKfB08gtMiRzYFI841lavNNogd95WdPKgcETbgzJeNWFKVKUZRDiqKcVRTljKIof52KwJLNmcvD2K1Z2CzTL1evL83lzQ5n5s2angdnOp1R73QmakaPSYbPHflju2qWODav0OYrv9Qcp899+lEwmDgU3gakp3ADDJftJq/vOFnebmoLrQhYlMFf8yba7j4m3H0JbnePkJ+leecTsBSCM0OEGwgB/1MIsQHYDXxaUZSNyQ0r+bzZ4Zg2246wviyXkbCa1MW2dMAdCHHZ6Y9rT8OqAgs6haT4q292anbNqlnsmqKcbMryTLzcHEf3YNAHpx+Byl281Guk1AwlS2S3m7niKt4JgK3juaT+AU0aMTYJTnS7ewRbtlZmOIGIVZIGE0JnFW4hRJcQ4vjo/13AWWBFsgNLJt6REK19nlkzyPWlmv/9+oXl7XM392ojU1cUzN6RkmXQUZWkBcq3OpyU5WWTkz37HtebKvJ49cIAofAsEwPP/B78DtR1N/Hq5VBalQFOJmBdQcBcir39IAUWIzazMakLxQknVru7NzlWid0EfV4xcR3EUgQhv9ZBucSZk8etKEoNsA14LRnBpIqGy8MImFW488xGSvOy0ytrSQKNo15xZRzCDdq6wenOxC9QvtnhoLZo5mw7wuYV+XgCYU51zPK7O/pTyFtBg34TjgBsLUpAoIuFouAu3k5+98voQz5qiqzp9dodvqxtEGwYU+k+n8BigOwEtbtHsGdDUAVHYNxr1Jo+JYFxC7eiKDnAI8BnhRBT/owrinKPoihHFUU52te3tNtGIy/meDzbqgILZ7vTKGtJAs29box6hdLc+GrkagotDHhG6HUFEhbDgDtAp8PPqlkWJiNsrMgDmNku6X4LOt6Ade/ipcvaOkZ9GswnmYnh4u3o1BHyu45QU2iludeNP5gmazTODrBO/AUkuoY7gn30pdzrnZRxQ1pUlsQl3IqiGNFE+yEhxKOxjhFC3C+E2CmE2FlcvLRf/acvD2MzGymwGGc9ttpuoa3fkz4v/iTQ2OOiwmZGp4tdgjeZKrsl+rxEEe/CZIQ8k5GaQgsvtcwg3Ed/ru1+suYdvNQZoionzeq3Y+C1rSdssGDvOEhNkYWwEAn9PSQVZ3sM4U5s12QE++g5ezyxmnCW/gJlPFUlCvBT4KwQ4jvJDyn5vNXhpKZo+tbt8VTZLahizOddjjT1uKi0xT9xacXosY09ifuZvdXhRGGsaiUeNlXkc+ziUOzZ1AEXvPkbqLmGEUMOb3SHqU9nmySCzoCraCsFHc9ROap4TQn8PSSVGBl3X4Lb3SOMZdzj1kAsdkDJGKvkauAjwNsVRTk5+u/mJMeVNPzBMM297rhsEtAyboBzCx3Qn6Z4AiE6HX5WxFFREiHfbCTXZKC5N3E/s1MdTipsZixZsy9MRti8Io9gWPB6rJkzr/wARtxwxX5O9ITxhdLc3x6Hq3g7xsAga0bOYdApNKVD0uF3QmB4asbtVZMi3BH7pWe8VaIzaAujaSDcs74LhBBHgPg+I6cBZ7uGCQsRt3CX5ZnI0us4v0x97sibPt6FSQBFUagsMNPYnTjBON3pjLmBw0xsLM8n26DjmYZu9q0bJwjuPnj5u1C9F4rW8dIbfnRAXYYIt7uwHqHoKLp8mHLbO2lKB6skUj89TrhHwgLnSOIrSgBMBrAatIx+ApbCtBDuZdc5ebZLexFPN+tiMjqdwooC87LNuCNv+rlYJQArbBYae10JqSxxeEfoHvaz0j63Aussg476ShtPn+lBHT8j/IV/hqAftv8FAC91hlljg5zZlzzSAtVoxZu/FtvlF1hhM6eHx+3s0G7HCXekJb0ggSNdx2M3TbJKIG26J5edcDf2uDAb9RTmxP9nvKrAzLmuNHjxJ4Gm0YqSkhgTFGeiqsCMyx+iZ3jhlSWRP5pVcxRugCtr7fS5Apzs0Jp3GGiBoz+DtTdCfiXDAcGp3gzxt8fhKazDOniG9bkjdAz5lv4elM527XaccEfb3ZO0YFyQDT2eGMKdKVUlmURTj4sVNhO6OBYmI1TbrfS5Awx6RpIY2dKkabSiRB9nRUmEiLXSlACfO7IBcPU8hHtblQ29TtF2zQmNwB/u1bzMrf8NgMdbgoQE7C1fcJhLCnfhFhQEu8UpBNDSt8R9bmeHtuvNhK5JTVSTUQ4IWsbdM9kqsRaO+u1L++e17IT7fI9rTgttAFV2TYTOLUOfu7HHFa0SmQuRn3EiKkvOdQ+Tm22Iq3xzMtZsA5sq8njqrS7Enz4PF1+CPZ+OCsR/nRuhJg/W5i84zCWFL28VIWMOG7xvAIn5A5pUnB1gKZ6wZVlfkromI0zbPQlLPuteVsI96Bmh3z1C1RyFO5LpnV9mPne0omQewp1vNpJnMiRkYexcl4squyWu8s1YXFlj5+3OR1GOPwB174dV1wNwfjDMqT6Vd1Zpoz4zCkWHx76Ziv6X0esSW5qZFJztY7vQjBKZJWJLlsedDYEwDI//IB0R7iW+QLmshHuurdsRIiK03Hzu1j5t78h4hkvFYkXBwhfGVFVwrsc1L38bQD8yzF193+Irxl/SmLMLtn04+thvzwcxKHB9Wk/emR534Ray/P1cm9u99Gu5Y9Rwd7tVco1aBUgyKIhZy50ebe/LSrgj2d9cRUBRFKrslmVnlTT3aT+vinmuDlUWWGjscS+osiSysBaxq+JFPzJM6flfUv/4u1hx8fc8ZnoPfz74ac4PabGMhAWPNga5qmxs89hMw124BYB3Zp1Z2pUlalgTyknC3eURFM39w17cFI7+3nvTsHtyWQl3Y48ba5Z+Xl5pVYGFxt6FiVC60dLrQadotezzobLAjDuwsMqSyJyYuEoBhSCv5zXWHPkf7Hz4Kla9/hVUvYkLu75K8a47MRqM/I/nfDgDgq++5GfQL7ixat6hLXlCJjv+nCp2hU/QPuhdumMbXN3azjeThdutJnUEQaR7smd8xm00QXbeWJXLEiVJH0KWJtrCpHleXumKAjO+kTCdDt+8rYN0o7nXTVmeCYN+fn/fK6Ot7y7K8uf3DoysK8z2M8/vOkLt61/BPHyBsMGCo/xahlZcjz+vVnsc+MwW+PpRlb0PufAE4T21sL1kXmGlDe7CLdReOkA2AZp73dFNJpYUMWq4Qcu4rypN3mULYg2aAsgtg8ELybtwAlg2GbcYHbYz14XJCGPlbUvcK0wgzX1uKuaxMBmhMgHDps53uyjLy8ZkjD3XUxd0s+rVL7Lx4F+gCwfo2PTfOf+279O14e6oaEfYUw43r9QWpb65F+7ZDHOsckw7PPZNGESQnbrGpTtvJ0YNtz8kGPQLipKYcVsMYDbEEu7yJS/cyybj7nMHcHiDc16YjFBp00SoqcfF9eszPE0DQmGVtn4PN9fNv8A5z6RNYFxI1+nZruFps21dyMfGg39JTv8p+le+m97VdyD0M5cgfHrLvENJS7y2K1AVA9fqTy/dksAYGXf3qO+cTI8btD/iU5pwcsqg7UUIB0G/NNtpl03GHVlVn6/NkWPS6oiXfFlVgrg06CWkigVl3KD9vOe7qOsPhmkb8MRsvFHUIOte+Aw5/ado33IfPes+NKtoL0dUgwmfbS37DGeW7mvX2QFZOZA19nvuGhXTZGbcoNklMa0SoYLjUnIvvgCWjXDPtxRwPBU2c3oM7EkA0e3Kpqko0YV8lJ/5MeVnfkxh2x/J8nbHPK7KbqGpx01YnfuibmOPC1XE7phc9eqXKOg8RNeGu3CV7przuZcTnoJNrBMX6OleopUSMUoBu9yacBenIOOeMq8kd/RT5tDStUuWjVXS2OMi12Qg3zz/jz6VBRZebOpDCDHvZpB0oWW0hjtWxp3l6WT9ob8iZ6ghel8oK4+z73gQd1H9hGOr7RYCIZW2AQ+rZ9nkdzINl0crSiZt6my/+CQlLb+lr/a9DFW+Y07nXI64CzdT0vowVc5j+IP7p10vWDRiNN90jVolyd7Ywm6CN3rFxPd0bpl2O9SW3IsvgGWUcbupnGdFSYQVNjPekTCXnf4ERrY0ae51U2AxTpl/bR14iy1/fA9mVxsXt36es9f9hNZdX0PVm9j4zIfJ7X1jwvEL6To9c3kYs1FPSd5YobUh4KD29a/gy62ld9Vt8/jOlh++vFWM6EzsVU5Hm6qWDEJoloR14rpRl1slNyt5zTcR7NngC4E7OO5Oi13ztpfwAuWyEG4hBOe7XQsu46sqGCtvy3RaYlSUKGqQNS/9DUJRaN31NdzF21CNFnz5a7iw8+8JZeez4eBfYnY0Rp+zwmZGp8xvI4qGy8OsLLRMGAi28tg/YQwMcnnTJybMtZDMgM7AUN4G9uqW4AKlp1/bQCGvYsLd3Z7kVpREKBx9iUesGQAUnWaXyIx7ceke9uMOhBbkb4NWyw3QvFQXeRKEECKmcJedexCLs4muKz7KiHXiGy1kstO244sInYFVr/6dtriDNhO7LN80540owqqgoXt4gk2S1/0qJS0P079yP/7cmvl9c8uUYPFmanU99FxsnP3gVDLQrN3mTZw7cNmtpkS4y0ZzuXbX5MqSUplxLzaNC6woiZBrMmIzGzM+4+5zBXD5QxOGS2V5u6k69W+4irbjLt4R83mh7AK6132YvL7jlDb9Knp/VYEluoFFvFwc8OAbCY9teCFUVh77J0ZMRfStet/cv6lljr9oMwCmjhcXOZJJDDRpt/mVE+7u8qhJLwUEKBm9RrsrRi330AXNylmCLA/h7l54RUmERAxOWuo0j85uHp9xrzz2TyhqiK4r/mLG5zrLr8Vt30T18W9GK02q7RbaB714R0Jxx3Bm0sJk4cUnyRk8Te/q22XZ3zwIWCtxKPmsGHpj9oNTSX8T6IxTmm+G/MkvBQRt1ne2HtqHJ1eWlEHQC56+5AcxD5aHcPe4sJmN5JkWXky/wmamKcNnloyVAmrCbRpupajtCQZW3kzQPEvzkaLQteFj6MIjVB/7BqCVBArmNlr0zOVhDDqFqgIzihqk6uS38edU4Sy/Zl7f07JHUbhg3kx98CSBYHD241PFQAvklU9Yr4jWcKcg41YUKDXHsEqiJYFtyQ9iHiwL4T7f46JyjtPlpqOyIPMrS851u8gZt3FB2blfoOoMDFTdFNfzRyxlDFT/GcVtf8A6cHpcZUn8PndD1zCVBWYMeh0lTb/B7LpIz5oPaAtHknkxZNtEkTJM5/njix3KGP2NU/ztLreWFCW7hjtCiSWWcI+WBC5Rnzvj3wWqKmjqdUdb1hdKZCTsua7MHfGqVeBopZP6kWFKWn7LcOkewtnxDyjqr7mVkDGH6hP/THFuNiajLm6fWwjBmU4nKwut6IIeKt/8Lh7bFbiLts73W5IAokTzub3nn13kSEYJh7SMdrJwp6hrMkKZBTpiLU6iyIx7seh0aPOcE+Fvw1hdcqbu+h4dxjX6fZY0/xf6kI+B6j+b03lUo4X+2vdi6zpCQdcRqu0W3upwxvXcPleAAc8INYUWys/+nCz/AD1rP5CB29SklkJ7Ea1qOZb2I4sdiobjIqjBKcLd7U5N802EUgu4RsAZGGd/6rO0pqAl2j2Z8cLdOM/NE6bDkmWgJDebsxmacXc5/bj8Ia1mXQ1Tdu5BPLYrpkzai4fBqncyYiqm+vg3WV9i5c1OB4HQ7DOhIwuT63MDrDjzI4aLd+KzrZvz9SUTMerhlH4TFc5j2sbJi02kFHBSRcllj0peCppvIpRGSgInL1DmLN3xrstAuCcutCWCaruFhgwV7kiHY5XdQkHnYUyezjln2xGEzkjvmveTM9TArcoLBMOC052zZ92nOhwowDXdv0AX8tGz5s55XV8ylYuWzZiEHzqPLXYo42q4JzXfuFPTfBNh2lru3DIYbEldIHNgVuFWFOVniqL0KopyOhUBJZrGHhd2axbW7MT9+a62W2jr9yzdHUUWwPmesY0LilsfJpiVj6t4+7zP5yzbizd/Ldde/D5WfBy7ODTrc15pGWCvfZjK5odwVLyNkZwM3RRyEfDaN6IKhUDjwcUORSsFzM4D08S1k7ZhNZoFp4LS6YTbVqWVA3oGUhdMnMSTcT8AzC/lWgI09rgS5m9HqC60oIrMbH0/3+2i0JqFDRcFHc/iLNsLugX80VMUutd/BFOgn/9leYKjbTMLtz8Y5sSlIb4s7tcy9tV3zP/akilUFOTwplhFcCksUA40TfG3Q6rg4rDKirnNI1sQOUbt3xSrpGCVdtuz9HLWWYVbCPECMJiCWBLOSEilsccVcyzoQoguUGbgru+RipKiC4+jU0M4KvYt+Jy+/DU4yq/hv6mP09PWMGMN/PFLQ7xbvMAVvuP0rLmTkKlgwdeXjLEqH55X67H2nwTvIr+t+5un2CTtLpWQCpXWaZ6TJEotMbonC2q023QU7nhRFOUeRVGOKopytK9vaXQbNfa4CIYFtUWJfRWU5pnINuiiG9lmCqGwSnOvmyq7heKWR/Dl1hDIrU7IuXvWfICwLot/CH2HSz3TC8bJ8618yfj/cOWtlSNbk4A9G47q6lEQ0PLc4gUScIG7G/InZtytDi3rrUxhxg3TNOGYbWC2Q8+Z1AYTBwkTbiHE/UKInUKIncXFxbM/IQWcuawthCVauHWKQpXdknGVJW0DXkbCKtuyu8kZfAtH+bUJO3fIZOetNf+dLboLqH/6fOyDgn72nvxb8hUvPRs/JpttkoCigDdvNcPkQPMi+tzRhcmJFSWLJtyjtdxTPg0WrITut1IbTBxk9DvjrU4nliw9pXmJX6JeadcGJ2VS63ukomSP+wBC0eMs35vQ85uqd/Aj9T3UXnoEXvnBxAE+oQDhX32ILYETPGz7eMIyfclUavJ1vBCuQzQ9A6o6+xOSQd957TZ/qnDnZ0FuisfRlFkgEIY+32ThroW+c1qz0BIis4W7w0lNoXXCPOdEUW234PQF6RkOJPzci8X5HhdGJczqridwFW0lnBV/p2Q86BQ4mH8Hr+u2wtP/G352EzQegNfuhwdvQd96kC+EPo6/euG+umR6VuXBc+F6FG8/dL+5OEF0HgeDacriZLMjtQuTESl0et8AABXdSURBVKat5S6ohfDI2BTDJUI85YC/Al4B1iuK0qEoyseSH9bCCYZVzna5qEmwTRKhenTcaCbZJQ2Xh7klp5Fsf19CbZLx1Bfr+ID3b+iu/4yWdf3nHfDk52HwAk9VfJqH1evZZE/KpSWjrMqHF9TRLeaan1mcIDqPQeGaKZthtDrUlNskML4kcLoFyqXlc8dTVfJBIUS5EMIohKgUQvw0FYEtlKYeNyNhlVXJEm67BQXNjskEhBCcbB/iDuOLhIw5uIu3JeU6b68EvU7HD9z74M/vh+u/DLf9FPW2n/KtwWtZbwPLwoc4SmZgRQ649Pl0Za+CpkUQ7tCI5hsXrp1wtzMgGPCLlFeUAJRbwaBA4+Ck3oz8Sq0cdon53BlrlZxO0sJkBEuWgcoCM8cvzd5Qkg50OnyMuIe40vcSzrI9CF1y1DM/G64ph0cag3gUM1RfBTmlPNkaosWhcuvcO+slc0SvwMpceFWph443Ul8W2NsA4QAUTRTuVocmmothlRh1UJULDQOTrBK9EfKrllxJYOYKd6cTs1HbNitZrCnJ5filIVQ1/RcoT7Y7eLf+VQwiiKM8uR7zzTXgCcJjTdpcaFUIvnc8QGUOXF0x83MliWFVHvzGu0PbYu7cH1N78cujY2WLJs6faXUuTkVJhNo8aBiI0Q1dUAPdUrhTwlsdo2NBkzhRbl1pDsO+EK396b8H5YlLDm7Xv4jPWjmvgVJzYUOBJhz/r2EEIQTPXgxxblDlzjVaNihJPusK4NWRWoKWUjj7h9RevPO41uqeUzrh7pYhFb0yNjsk1azKg16vYMAXY4HS3b2kWt8zUrhDYZWzXcNJs0kirC3NBeD4RUdSr5MKeltPs0PXiLPi2qSPT1UUuHml9rF0xy/c/O3zfsqtsE+OJEkZm+0ACs25u6DlEPhTuFbTeVxbmJz0Omt1qprXvEiqVJun3Z6dbJfYRxOZnqXjc2ekcDf2uPGH1KQLd3m+iZxsQ9r73MGwytb+xwijx1H+tpRc88aVcO8W2FEsKDIJPrYB9Bn5alyaVFjBlg0HxFXaTOzGp1Nz4RGPVhddNHVMb4tDZcUiLExGGBPuSXZJ4VpAgUuvpjym6cjIt8orrdpHmo3leUm9jk5RWFOSw9E4Jt4tZc53DPAe5QUu5m6f0y43C0GvwJ+thM9uhX+9FvaUp+SyklEUBTbZ4bcDtWAphIbHUnPhrjdBhKcsTEaGSy2Wvw3awnmRKcYCZXaO9gmh9fCixBWLjBTul5v7Kc83UZiTnfRrrS3JobnXjdO3hDZgnSMDx39HkTKMq+rtix2KJIVsskOHR8Fdvltrfx/xJP+i0yxMnhtUGQlrPvNiUjPdAmVZHXQcTc3PKA4yTrhDYZVXLwwkPduOsG7U5z6RxnZJWfNv6BJF6MvrFjsUSQqJNDqdNO2GkD81dknnMbAWg3ni1Mej3ZpYblzk5qtVedoiaSA8qVKsvF6zlC69sjiBTSLjhPutTieeQJhNFfF/5Df4Byhq/R1lZ39OUevvye15XSuTioPVxTnoFDh+KU0XKIfaWO85youm61B0GfdykMxAbT5YDPCUd60mpicfSu4F1bC2EFqyccpDR7tDFJm1HdcXk1X5EBLQNDTp/V+ySWvEufDC4gQ2iRTt6pY6Xm4Z9bcrZs+4c/qOs/L4N8ntO4YySah9ebV0bbib3tW3I/TTWy7mLD3VdguvtS6dUqG5MHzkx1iFQlfpPjYtdjCSlKJXYIMdXusC1t8Ap34NjktgS9KAr/bXwTcI1bsn3C2E4I2uMBuXwOj1iFXT0B9mc9G4dnyjCYrXLxnhzrgU6+WWfqrtFvLN03f+6YJual7/KpufugPTcCt9te+l5aqvc27fj2ja+y90bP4UoLDqtS9T9+SfYxpunfGaW6tsHG0bwuFdAhuwzgX/MKZTD/CkehUbVhQudjSSRWCTHZocKs6q0dnnx3+ZvIs1PqllrRUTt8LrdAt6vGLRbRKAMiuY9DEWKAHKtsDlk+BbfFs0o4Q7EApztG1oxmw7y3OZuidvo+z8LxmsupGWPf9M3+rb8eetIpyVy4i1HGf5NbTu+gcu1f9Pst2dbPnjrRS2PTHtOXestBMWgkPne5PxbSWP4w+SFXLzO+P+RWkzliw+daN/r18YKoQVO+DEL5M3wvTcn7RFvqyJNX/Hloi/DdqnkDU2eL0rxs+gvB4Q0PZSyuOaTEYJ94lLDgIhlU3TCLdl6Bybn7yNbE8nF7d/ge4r/hLVMM1+lIqCq2QHLbv/CX9OJetevI+yc7+IeeiqYisFFiPPNPQk6ltJPqER1Fe+zyvqRvLKVi12NJJFYn0BFGTDkxeCsPZGcHUlZ2Jgf7M2GrVy15SH3ugOYTFATW7iLzsfdpZoGXe3Z1LWXbQe9Nlw4fnFCWwcGSXcBxt60OsUNpRNFW7rwFtsevpOdGqQCzv/Hk9hfBUUIVMhF3d8keHindS+8VUqTv/fKcfoFIXt1QUcPt9HIJQmO7+ffgSdq4sfhm7hytLZD5dkJnoFri6H5y6G8JRdqW3VdfRnib9Q45PabdVU4T7aHeb/b+/Mo6Oq8jz++b3KSgxJIAaTEAgJARKYyBIEUUDcm9amcZkGu2GYaYe2bZexER2UOXp0tHvsaRltmXG6sQ8OHKVtunVcUHaaBolsIWzKEkhCAhIICUlIUqnlzh+vcJIikJeQqleF93NOznmv6la9b93c+tV9v3t/v9/gpNAJwLrO931YX+4363ZEQtq1ZooAmwsrhEhXXT4uj5cPdlUyIiORuOi2a649ar4kd81MvI4Yjo5+rtPVVZQRybH8x6i9Zhz9i35F+p6FF7QZ1T+JxhYPW0rCYJHS44ZNCzgR2Y9tRr7Of/0tZ3waNHtg3TFg8Hfg0Co4XtS9Fzmwwsz54ZefpM6pOHDGGxILk+fpd5VZg3JtWTvGeeBtUP+1fXnMfVwxhnvjwVOcbmhh4qC29S5jzx4mb/UMEAelo57BFdvFephGBJXDHqY29Ub67fo1qfsXtXl6aFoCMZEGa74MA3fJzrfh9AFec9/DqBSxLTeEJjTI7QW9YuCTIy7InQLR8bDupe67QEMVlH/R7mx76wk3yqchVBCB0X1gc6WbZrfffu6+o8096DsW26LtPFfMV/ZPOyvoGRPB8IzEbx6LqTtK3uofAl5KRz1rZkK7HMSgMu8nnO0zlswdL7fxeUdFGOSnJ7Jq30ncHpvq+FmhqRbW/yt1SUNZ1jSa61LsFqSxm/PukvXlbhqIhWH3mjPK7srNUfifZlxE1oWRue8fctEz6nzSq9BhdAo0uaHwuJ/r04iA7FvMu5K64/aI4wox3LWNLazef5JxA5OJ8DnKouuPkbf6hxhuJ2Ujn6ElrpuSYRgOKoY9/I3PO+XQsm+eGp+TTFW9kxV7v+6eawWCv7wCjTW8bswkPlIYp3OEaIDxqWax3LVlbhh8lzmrXPti24LOXaGpFrb+DjJvhIS26R/PNHlZWepmUjpEOi7yepvITza3Ba4vbyeVRc7t5g9RoAOWLsEVYbg/Kj6Oy6OYkGO6QaLryxi6ajoOVwOlo+bhvKpvB+/QSYwIKvIfpT55OFmFz3J1yZ8AGNk/ifTEWN7cUBKa1d+rvoSt/019/9t4qzKDyZkQc8WFYGm6Qm4vs+7iW3ucqIho+Jv7oWzT5Sef2rYIWhrM9/Pj/UMu3F64PUDxPpdDlMM03qvL3Hj8C6X0TDP3dO9cAl577q7D3nA73R5+v7mU/r17kNm7BzFnSxi28gc4XPWUjZyHM75/QK5rLlj+E+d6DSX786dIOfgOhgjfzU9l/4k6/nrodECu22WcDfDeTIi6ioXqfiIMuCvTblGaUMEQmJYDu095WVXqhsGTzURQHz0OZyu69qYtjVC4ENILoFfbLadKKf7wlYtBiWZip1Dk1r5wvEHxweF2Zt2DJ0NtGewIwA4cC4S94X5r01GOnj7HtNH9iK/ezdBV0xCPk9KC+QGv5KIcUZQPn0ND8nCyv5hP+u43uDG7N73ionjzLyUBvXanUMr8AlYfpm7skywuiWNSX3NBSqM5zy19oW8c/HqbEw8OGP+kWRvyz7PNPCOdpXChWc+yndl28SkvB2u83J7RDcIDxPWpMDABFmx30uKfdKr/DZA6HFY/Z4uvO6wNd2VtE79Ze5jRmUnc6trA0JU/ADEoLZiP86rgjAjliKb82ifM3SbFrzJk6zzuzkvi85JqNoXKrLvwv2Dvchj+I16tGEKzB6bqmBuNHw4DHhgMB2u8fFziNl0CYx6Css2w5vnO+bsPr4X1L0PmeOjTNguOUoo3dzmJdsCEEK56ZAjMGAwV9ebdQRtEYOzPwOOCFXODry3oV+wmlFK88NE+olQT/xa7hJzNP6cpIZsj171IS1yQR4MRQeXQh6gaMJWUkuXMLX+YsQlneHxZEV+fbQ6ultYoBRv/HVbOg37X81ncFBbvbeHuAdAvRKLUNKHF+DSzEszLhc1m5GDWzTDoO/D56/DJz63NvM8cgeX/YCarGvfYBU+/f8jFZ0fdTMuBuIunFAoJRqWYofi/2enknMvf150Kw6fDVx/DrneDqissDXeL28ucPxZz7ss1rI+dR9bRd6jOuIOykfPwRNnkMBODUwPvp2zEU8Q0VrLUNYeZrj/w+NIttLhtWMBwO+HTp2Hdi5B1E+Uj5jJ3o5NBifDj3ODL0YQHhsATw6GuRTFrRSP1LmDswzDsPjOi8r2Zl/Z5l26GJVPNXReT5kNk25QSx+q9/MumZob2gnsHBvazdAciMGsInG5UzPjkHGedfsY7byr0GQYf/NTcPRMsXVZ2P4jIncBrgANYpJT65aXaFxQUqO3bt3ePwlZ4vYrdFTWs/PBdbqr6H8YYX+HscQ3H82bTmDSk26/XVSKaq7nm4FISTn5BufdqNiQ/wF0znqBXUpDCw45sgE/mQPVhyJ3C7sxZPLK2mTNNitcnmLsHNJpLsbMKnt8KY1IdLLwtlqQYA/Z9ADsXgzig4O/NBbqUPNO6nSiGfX+GoqVmdOT4JyGl7Qyhst7LT1Y1cqTWyxsTw2scbj4Bv9oJWYkGSyb3ICWu1ZzX3QwbXzHT1k54CiY906WC2yKyQylVYKltR4ZbRBzAQeA2oALYBkxXSu2/2Gu6bLi3LYL4NEjNh57p5of3uKG2jFUbNlCzdxUTvFtJlTPUR/SmIfu71KTfjHJEdf5aQSCuei/R+94l1XmUOuKoGXA36WO+T0TWRIjq5lHrbDC3bhUthfLPIT6VcyNms7g2nwXbnSRGw9MjQyMDmyY8WHcMFhRDzyh4ZmwM9wyKJKLxFOxeZvqwlZ/bRByQNwWufcDMX+3DqxQfl7iZ/9cmXF6YOwLGXBPkD9MNFJ2Cl7bBtNwonrvBb2Xf64Etb8DZcnhwnVmnspN0t+G+HnheKXWH73wegFLqFxd7TZcMt8cFL6eBx5fT2vBtMPZ6AFOjkygq4q/FkzYad9+xKCMMNiErRU3lQZq/WsX13p3EiRM3EdTFZ+NOGYbROxsjKYOonn2IjksgMjbeTHsZGQeGw7zlVF6zH5THNNDOOmg4ibemDFV9BKnYhlTtQ5SHxtg09iTdwhLvHawsN3B5YVwqPJYP8aH5+6YJYUrrYOEe2H8GEqJgUr9IClIdZEfX0c9dRnxTOVEGqKQsvL2yaDbiaHRBVaOXsjovxVUePj3qpqpRMSgRnhoJqTZWcr9cTjYZ3JAZR7SjnRm1UmZR4YS0Lr13dxvu+4A7lVIP+s5nAGOUUo9c7DVdnnE3VJl176oPw7kqwADDgPhUSMrkGH1oJjytj9ur2FnZxPHDxSTW7GawKiXXKOdqOXtZ71uvYtntzaJIDWS9Zzg71CBASI4VxmdEcmtmJDlJBtKFWzeNBswZc2Glm00VbraecF/o570EUQ4YnRrBhIxIJmREEGGE9ziMiXSQkXSRVNAAV+e2udvoDJ0x3FamrO319AX/ORGZDcz2nTaIyAErAgJAMhAi+/A6RRd11wEngS3A/1cvKQN2AP/RLdIuybesv20n7HQfAt4JQ90+gqnbcrSgFcNdAbTeFN0XuGDHuVLqt8BvrV44UIjIdqu/WqGE1h1ctO7gonV3L1a2A24DckRkgIhEAdOADwMrS6PRaDQXo8MZt1LKLSKPACsxtwP+Xim1L+DKNBqNRtMulrZlKKVWACsCrKW7sN1d00W07uCidQcXrbsbsRSAo9FoNJrQISxD3jUajebbTNgabhG5U0QOiMhhEfnndp5/SET2iMguEdkkInl26PSnI92t2t0nIkpEQmJF20J/zxKRU77+3iUiD9qh0x8r/S0ifysi+0Vkn4i8E2yN7WGhvxe06uuDIlJrh05/LOjuJyLrRaRIRHaLyGQ7dPpjQXd/EVnr07xBRLq5OksnUUqF3R/mImkJkAVEAcVAnl+bnq2Ovwd8Fg66fe3igY1AIVAQDrqBWcAbdmvtgu4coAhI8p2nhINuv/aPYm4aCHndmD7jn/qO84DSMNH9R+DvfMc3A0vs1ByuM+7rgMNKqSNKqRZgGTCldQOlVF2r0zjaCRqygQ51+3gReAWwMSdsG6zqDjWs6P5HYKFSqgZAKVUVZI3t0dn+ng4EN69o+1jRrYDzKTwTaCcmxAas6M4D1vqO17fzfFAJV8OdDhxrdV7he6wNIvIzESnBNIIXJgYOPh3qFpERQIZS6uNgCusAS/0N3Ou7lVwuIqFQ28SK7kHAIBHZLCKFvkyYdmO1vxGR/sAAYF0QdHWEFd3PAz8SkQrMnWqPBkfaJbGiuxi413c8FYgXkd5B0NYu4Wq4LYXhK6UWKqWygaeB+QFX1TGX1C0iBrAAmBM0Rdaw0t8fAZlKqXxgDfB2wFV1jBXdEZjukpswZ66LRCQxwLo6wtL49jENWK6Uf6o+W7CiezqwWCnVF5gMLPGNezuxovtJYKKIFAETgUrAHWhhF8PuDusqlsLwW7EM+H5AFVmjI93xwDBgg4iUAmOBD0NggbLD/lZKVSulnL7T3wGjgqTtUlgZJxXA/yqlXEqpo8ABTENuJ50Z39MIDTcJWNP9Y+A9AKXUFiAGMx+InVgZ38eVUvcopUYAz/oeu7wMcZeD3QsDXVxMiACOYN4inl9MGOrXJqfV8d3A9nDQ7dd+A6GxOGmlv1NbHU8FCsNE953A277jZMxb5t6hrtvXbjBQii8ew+4/i/39KTDLd5yLaSBt1W9RdzJg+I5fAl6wVbPd/+zL6OzJmAUeSoBnfY+9AHzPd/wasA/YhbmYcFEDGUq6/dqGhOG22N+/8PV3sa+/h9it2aJuAV4F9gN7gGl2a7Y6TjD9xb+0W2sn+zsP2OwbJ7uA2+3WbFH3fZiJDg8Ci4BoO/XqyEmNRqMJM8LVx63RaDTfWrTh1mg0mjBDG26NRqMJM7Th1mg0mjBDG26NRqMJM7Th1mg0mjBDG26NRqMJM7Th1mg0mjDj/wCYo93V4GnuKAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.kdeplot(standard_unq_array, shade=True, label='Random Sampling')\n",
    "sns.kdeplot(seq_unq_array, shade=True, label='Sequential Sampling')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can compare average label uniqueness using sequential bootstrap vs label uniqueness using standard random sampling by setting compare parameter to True. We have massively increased the performance of Sequential Bootstrapping which was described in the book. For comparison generating 50 samples from 8000 barrier-events would take 3 days, we have reduced time to 10-12 seconds which decreases by increasing number of CPUs.\n",
    "\n",
    "Let's apply sequential bootstrapping to our full data set and draw 50 samples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Standard uniqueness: 0.9465875370919882\n",
      "Sequential uniqueness: 0.9913169319826338\n"
     ]
    }
   ],
   "source": [
    "bootstrapped_samples = seq_bootstrap(triple_barrier_ind_mat, compare=True, sample_length=50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes you would see that standard bootstrapping gives higher uniqueness, however as it was shown in Monte-Carlo example, on average Sequential Bootstrapping algorithm has higher average uniqueness."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have understood the concept of concurrency and how it can make your data set 'impure', how Sequential Bootstrapping can help us to tackle concurrency proble. The next step is to combine standard Random Forest with Sequential Bootstrapping and show how SequentiallyBootstrappedRandomForest increase quality of our machine learning models."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
